WHAT IS CLAIMED TS; 

1. A computer- readable recording medium with video index 
information prepared by analyzing the video information 
recorded therein, 

the video index information having a data structure 
wherein the video index information has a tree structure 
comprising, as structure element objects, at least one frame 
information for managing a given number of successive frames 
in the video information as one linit of information, at least 
one sound Information for managing voiced corresponding to a 
given range in the video information as one unit of Information, 
at least 0x10 segment information for classifying /£he frame 
information and sound Information each as one unit of 
information and managing the information as a desired 
significant collection of data, and package Information for 
collectively managing a plurality of segment information as a 
group ; 

further the segment information manages the package 
information for collectively managing a plurality of other 
segment information generated by using the same frame as that 
of the frame information and sound information managed by the 
segment information and voices in the frame; and 

the package information is arranged, in addition to the 
frame information and sound information, under one segment 
information within said tree structure. 
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2. The recording medium with video index information 
recorded therein according to Claim 1; wherein, of the structure 
element objects, at least one type of structure element object 
has retrieval information for retrieving contents of frames 

5 and/or voices directly and indirectly managed by the structure 
element object. 

3. The recording medium with video index information 
recorded therein according to Claim 1; wherein the structure 

10 element object has retrieval information for retrieving frames 
and/or voices directly or indirectly managed by each structure 
element object. 

4 . A video information management method which uses a video 
15 index information for managing the video information by 

analyzing at least video including a plurality of frames, 
generating, apart from the video information, video index 
information for managing the video information, and referring 
to the generated video index Information; wherein 

20 the video index information has a tree structure 

comprising, as structure element objects, at least one frame 
information for managing a given number of successive frames 
in the video information as one unit of information, at least 
one sound information for managing voices corresponding to a 

25 given range in the video information as one unit of information, 
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at least one segment information for classifying the frame 
information and sound information each as one unit of 
information and managing the information as a desired 
significant collection of data, and package information for 
5 collectively managing a plurality of segment information as a 
group ; 

further the segment information manages the package 

information for collectively managing a plurality of other 

segment information generated by using the same f rame as that 
10 of the frame information and sound information managed by the 

segment information and voices in the frame; 

the package information is arranged, in addition to the 

frame information and sound information, under one segment 

information within said tree structure; and 
15 the video information is managed by using said tree 

structure and structure element objects of the video index 

information. 

5, The video information management method which uses the 
20 video index information according to Claim 4; wherein a 
particular structure element object in said tree structure of 
video index is specified to make the specified structure element 
object visible and/or audible. 

25 



104 



6- The video information management method which uses the 
video index information according to Claim 4; wherein the frame 
information has a node ID in said tree structure, a range of 
successive frames managed according to the frame information, 
5 and a pointer indicating a position in the video information. 

7. The video information management method which uses the 
video index information according to Claim 4; wherein the sound 
information has a node ID in said tree structure, a given range 

10 to be managed according to the frame information, and a pointer 
indicating a position in the video information. 

8. The video information management method which uses the 
video index information according Claim 4; wherein the segment 

15 information has a node ID in said tree structure, identifying 
information for identifying frame information, sound 
information and package information to be managed according to 
the segment Information, and a pointer to upper package 
information with the segment information belonging thereto. 

20 

9. The video information management method which uses the 
video index information according to Claim 4; wherein the 
package information has a node ID in said tree structure, a list 
of segment information managed according to the package 

25 information, and a pointer to upper segment information with 

105 



the package information belonging thereto. 

10. The video information management method which uses the 
video index information according to Claim 4; wherein the video 

5 index information has a given attribute object (106) allocated 
in said tree structure, all of structure element objects in the 
video index information each have a pointer to the attribute 
object (106), and given additional information can be added to 
a given structure element object using the attribute object 
10 (106) and the pointer to the attribute object (106). 

11. The video information management method which uses the 
video index information according to Claim 4; wherein the video 
index information is separated from the video information and 

15 is stored in the separated state. 

12. A video information management method which uses a video 
index information for managing the video information by 
analyzing video information including at least a plurality of 

20 frames, generating, apart from the video information, video 
index information for managing the video information, and 
referring to the generated video index information; wherein 
the video index information has at least a plurality of 
segment information for managing a range of a given number of 

25 successive frames, and a link list for indicating links between 
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segment information for indicating in what order the plurality 
of segment information should be followed; 

a plurality of link lists each indicating in what a 
desired order of segment information are to be followed of the 
5 plurality of segment information each constituting the video 
information, a plurality of view information corresponding to 
the link lists are set, and the video information is managed 
by using the video index information having the plurality of 
view information. 

10 

13. The video information management method which uses the 
video index information according to Claim 12; wherein, of the 
plurality of view information in the video index information, 
particular view information is specified, and a portion of the 

15 video information is made visible and/ or audible by using a 
link list for the specified view information. 

14. The video information management method which uses the 
video index information according to Claim 12; wherein the video 

20 index information is separated from the video information and 
is stored in the separated state. 

15. A computer- readable recording medium with audio index 
information prepared by analyzing audio information recorded 

25 therein; wherein 
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the audio index information has a tree structure 
comprising at least one sound information for managing voices 
corresponding to a given range in the audio information as one 
unit of information, at least one segment information for 
5 classifying one the sound information equivalent to one unit 
of information and managing the sound information as a 
collection of information having desired meaning, package 
information for collectively managing the segment information 
as a group each as a structure element object; 

10 the segment information manages package information for 

collectively managing a plurality of other segment information 
generated by using the same range of sounds as the sound 
information managed according to the segment information and 
managing the segment information as a group; and 

15 the package information is allocated in addition to the 

sound information, under one segment information within said 
tree structure. 

16. The recording medium with audio index information 
20 recorded therein according to Claim 15; wherein, of the 
structure element objects, at least one type of structure 
element object has retrieval information for retrieving 
contents of sounds directly or indirectly managed by the 
structure element object. 

25 
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17. The recording medium with audio index information 
recorded therein according to Claim 15; wherein the structure 
element object has retrieval information for retrieving 
contents of sounds directly or indirectly managed by each 

5 structure element object. 

•i 

18. An audio information management method which uses an 
audio index information for managing the audio information by 
analyzing audio information, generating, apart from the audio 

10 information, audio index information for managing the audio 
information, and referring to the generated audio index 
information ; wherein 

the audio index information has a tree structure 
comprising at least one sound information for managing sounds 

15 corresponding to a given range in the audio information as one 
unit of information, at least one segment information for 
classifying the sound information equivalent to one unit of 
information and managing the sound information as a group having 
desired meaning, package information for collectively managing 

20 the group of segment information each as a structure element 
object; 

the segment information has package information for 
collectively managing a plurality of other segment information 
generated using sounds in the same range as that of the sound 
25 information managed by the segment information as a group and 
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managing the group of information; 

the package information is allocated, in addition to the 
sound information, under one segment information within said 
tree structure; and 
5 the audio information is managed by using said tree 

structure of the audio index information and the structure 
element objects. 

19. The audio information management method which uses the 
10 audio index information (100) according to Claim 18; wherein 

the audio index information (100) is separated from the audio 
information and stored in the separated state. 

20. A video retrieval method which uses a video index 
15 information for retrieving video information satisfying 

desired retrieval conditions from a plurality of video 
information by referring video index information prepared in 
correlation to each video information; wherein 

the video index information has a tree structure 

20 comprising at least one frame information for managing a given 
number of successive frames in the video information as one unit 
of information, at least one sound information for managing 
sounds corresponding to a given range in the video information 
as one unit of information, at least one segment information 

25 for classifying the sound information equivalent to one unit 
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of information and managing the sound information as a group 
having desired meaning, package information for collectively 
managing the group of segment information each as a structure 
element object; 

5 at least one type of structure element object of the 

structure element objects has retrieval information for 
retrieving contents of a frame and/or sounds directly or 
indirectly managed by the structure element object; 

the segment information manages package information for 
10 collectively managing a plurality of segment information 
prepared by using the frame information managed by the segment 
information as well as a frame and sounds in the same range as 
that of the sound information and managing the group of segment 
information; 

15 the package information is located, in addition to the 

frame information and sound information, under one segment 
information within said tree structure; and 

retrieval conditions for desired vide information are 
inputted, a plurality of video index information prepared from 

20 a plurality of different video information to identify video 
index information including retrieval information satisfying 
the retrieval conditions, and a list of video information 
corresponding to the identified video index information is 
outputted as a result of retrieval. 

25 
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21. The video retrieval method which uses the video index 
information according to Claim 20; wherein the frame 
information further has retrieval information for managing a 
representative frame freely selected from successive frames in 

5 the video information managed as one unit of information and 
retrieving contents of the representative frame; and 

the sound information has retrieval information for 
managing a representative noise and sound in a given range 
selected from sounds corresponding to a given range in the video 
10 information managed as one unit of information and retrieving 
contents of the representative voice and sound. 

22. The video retrieval method which uses the video index 
information according to Claim 20; wherein further a hierarchy- 
is level of the video index information is inputted as a retrieval 

range together with the retrieval conditions, video index 
information including retrieval information satisfying the 
retrieval conditions is identified by retrieving only retrieval 
information for structure element objects present in the 
20 retrieval range in the plurality of video index information, 
and a list for video information corresponding to the identified 
video index information is outputted as a result of retrieval. 
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23. The video retrieval method which uses the video index 
information according to Claim 20; wherein only retrieval 
information in video index information prepared in correlation 
to video information is used when retrieving desired video 

5 information from the plurality of video information. 

24. The video retrieval method which uses the video index 
information according to Claim 20; wherein the retrieval 
conditions include a plurality of retrieval items each for a 

10 specified type of structure element objects. 

25. The video retrieval method which uses the video index 
information according to Claim 20; wherein the retrieval 
conditions include a plurality of retrieval items each for the 

15 same type of structure element object. 

26. The video retrieval method which uses the video index 
information according to Claim 24; wherein the retrieval 
conditions can logically express relations among the plurality 

20 of retrieval items. 

27. The video retrieval method which uses the video index 
information according to Claim 24; wherein the retrieval item 
has the same data structure as that of the retrieval information 

25 present in a structure element object as an object for 
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retrieval. 

28. The video retrieval method which uses the video index 
information according to Claim 25; wherein the retrieval item 

5 has the same data structure as that of the retrieval information 
present in a structure element object as an object for 
retrieval . 

29. The video retrieval method which uses the video index 
10 information according to Claim 20; wherein a list of structure 

element objects having retrieval information satisfying the 
retrieval conditions is outputted together with the list for 
video information for each video information on a list outputted 
as a result of retrieval. 

15 

30. The video retrieval method which uses the video index 
information according to Claim 20; wherein the retrieval 
conditions comprise a plurality of retrieval items capable of 
being weighted independently, and a score assessed according 

20 to weighting of the retrieval items is given to each video 
information on the list outputted as a result of retrieval. 

31. The video retrieval method which uses the image index 
information according to Claim 30; wherein each video 

25 information on the list outputted as a result of retrieval is 
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outputted according to the assessment value in the ranking 
order . 

32. A video retrieval method which uses a video index 
5 information for retrieving a video satisfying desired retrieval 
conditions from image information by referring video index 
information previously prepared from video information; 
wherein 

the video index information has a tree structure 

10 comprising at least one frame information for managing a given 
number of successive frames in the video information as one unit 
of information, at least one sound information for managing 
sounds corresponding to a given range in the video information 
as one unit of information, at least one segment information 

15 for classifying the frame information and sound information 
each as one unit of information as a group having desired meaning 
and managing the group of information, and package information 
for managing a plurality of segment information as a group each 
as a structure element object; 

20 the structure element object has retrieval information 

for retrieving contents of frames and/or sounds directly and 
indirectly managed by the structure element object; 

the segment information manages package information for 
managing a plurality of other segment information generated 

25 using frames, sounds in the same range as that of the frame 
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information and sound information managed by the segment 
information; 

the package information is located, in addition to the 
frame information and sound information , under one segment 
5 information within said tree structure; 

retrieval conditions for a desired video -image are 
inputted, a structure element object including retrieval 
information satisfying the retrieval conditions is identified 
by retrieving the video index information, and a list for the 
10 identified structure element objects is outputted as a result 
of retrieval. 

33. The video retrieval method which uses the video index 
information according to Claim 32; wherein a hierarchy level 

15 in a direction of depth in said tree structure is inputted as 
a retrieval range together with the retrieval conditions, a 
structure element object including retrieval information 
satisfying the retrieval conditions is identified by retrieving 
only retrieval information for structure element objects 

20 present in the retrieval range in the video index information, 
and a list of identified structure element objects is outputted 
as a result of retrieval. 
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34. The video retrieval method which uses the video index 
information according to Claim 32; wherein the retrieval 
conditions comprises a plurality of retrieval items capable of 
being weighted independently, and a score assessed according 

5 to weighting of the retrieval items is assigned to each 
structure element object on the list outputted as a result of 
retrieval . 

35. The video retrieval method which uses the video index 
10 information according to Claim 34 ; wherein each of the structure 

element objects on the list outputted as a result of retrieval 
is outputted according to the assessment value in the ranking 
order . 

36. An audio retrieval method which uses an audio index 
information for retrieving audio information satisfying 
desired retrieval conditions from a plurality of audio 
information by referring to audio index information previously 
prepared in correlation to each audio information; wherein 

the audio index information has a tree structure 
comprising at least one sound information for managing sounds 
corresponding to a given range in the audio information, at 
least one segment information for classifying the sound 
information as one unit of information and managing the sound 
information as a group having desired meaning, package 
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information for managing the plurality of segment information 
each as a structure element object; 

at least one type of structure element objects of the 
structure element objects have retrieval information for 
5 retrieving contents of sounds directly or indirectly managed 
by the structure element object; 

the segment information manages package information for 
collectively managing other segment information prepared by 
using sounds in the same range as that of the sound information 
10 managed by the segment information and managing the segment 
information as a group; 

the package information is allocated in addition to the 
sound information, under one segment information within said 
tree structure; and 
15 retrieval conditions as conditions for desired sounds are 

inputted , audio index information satisfying the retrieval 
conditions is Identified by retrieving a plurality of audio 
index Information previously prepared from a plurality of 
different audio information , and a list of audio information 
20 corresponding to the identified audio index information is 
outputted as a result of retrieval. 

37. The audio retrieval method which uses the audio index 
information according to Claim 36; wherein positional 
25 information identifying a position of a desired voice is added 
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according to a structure element object satisfying the 
retrieval conditions to each audio information on a list 
outputted as a result of retrieval and the audio information 
with the positional information added thereto is outputted. 

5 

38. The audio retrieval method which uses the audio index 
information according to Claim 36; wherein the retrieval 
conditions comprises a plurality of retrieval items each 
capable of being weighted independently, and a score assessed 

10 according to weighting of each retrieval item is added to each 
audio information on the list outputted as a result of 
retrieval . 

39. The audio retrieval method which uses the audio index 
15 information according to Claim 38; wherein each of audio 

information on the list outputted as a result of retrieval is 
outputted according to the assessment order in the ranking 
order . 

20 40. An audio retrieval method which uses an audio index 
information for retrieving sounds satisfying a desired 
retrieval condition from audio information by referring to 
audio index information previously prepared from audio 
information ; wherein 

25 the audio index information has a tree structure 
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comprising at least one sound information for managing voiced 
and sounds corresponding to a given range in the audio 
information as one unit of information, at least one segment 
information for classifying the sound information as a group 
5 and managing the sound information as a group having desired 
meanings , and package information for collectively managing the 
group of segment information; 

•. the structure element object has retrieval information 
for retrieving contents of sounds directly or indirectly 

10 managed by the structure element object; 

the segment information has package information for 
collectively managing a plurality of other segment information 
prepared using sounds in the same range as that the sound 
information managed by the segment information as a group; 

15 the package information is allocated in addition to the 

sound information under one segment information within said 
tree structure; and 

retrieval conditions for desired sounds are inputted, a 
structure element object including retrieval information 

20 satisfying the retrieval conditions is identified by retrieving 
the audio index information, and a list for the identified 
structure elements is outputted as a result of retrieval. 
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41. The audio retrieval method which uses the audio index 
information according to Claim 40; wherein a hierarchy level 
in a direction of depth in said tree structure is inputted 
together with the retrieval conditions, a structure element 

5 object including retrieval information satisfying the 
retrieval conditions is identified by retrieving only retrieval 
information for structure element objects present in the 
retrieval range in the audio index information, and a list for 
the identified structure element objects is outputted as a 
10 result of retrieval. 

42. The audio retrieval method which uses the audio index 
information according to Claim 40; wherein the retrieval 
conditions consist of a plurality of retrieval items each 

15 capable of being weighted independently, and a score assessed 
according to weighting of the retrieval item is given to each 
of the structure element objects on the list outputted as a 
result of retrieval. 

20 43. The audio retrieval method which uses the audio index 
information according to Claim 42; wherein each of structure 
elements on the list outputted as a result of retrieval is 
outputted according to the assessment value in the ranking 
order . 

25 
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44. A video retrieval system comprising: 

a plurality of video information databases each storing 
therein a plurality of video information; 

a plurality of video index information databases each 
5 storing video index information previously prepared in 
correlation to each video information; 

a retrieval information delivery file generated by 
extracting retrieval information for retrieving video 
information stored in said video information databases from the 
10 video index information stored in said plurality of index 
information databases; and 

a retrieval device for retrieving video information 
completely or almost satisfying a desired retrieval condition 
from said video information databases by using said retrieval 
15 information delivery file; wherein 

the video index information in said video index 
information databases has a tree structure comprising at least 
one frame information for managing a given number of successive 
frames in the video information as one unit of information, at 
20 least one sound information for managing sounds corresponding 
to a given range in the video information as one unit of 
information, at least one segment information for classifying 
the frame information and sound information each as one unit 
of information and managing the information as a group having 
25 desired meanings, and package information for managing a 
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plurality of segment information as a group each as a structure 
element object; 

of the structure element objects, at least one type of 
structure element objects have retrieval information for 
5 retrieving contents of frames and/or sounds directly or 
indirectly managed by the structure element object, the segment 
information manages package information for managing a 
plurality of other segment information prepared by using the 
frame information managed by the segment information and sounds 

10 in the same frame as that of the sound information as a group, 
and further the package information is allocated in addition 
to the frame information and sound information under one segment 
information within said tree structure; 

said retrieval information delivery file stores therein 

15 in a correlated form the retrieval information extracted from 
said plurality of video index information databases, and the 
retrieval information classified into a plurality of categories 
with a structure element object as an extraction source of the 
retrieval information and video index information having the 

20 structure element object present; and 

said retrieval device identifies, when retrieval 
conditions for a desired video-image are inputted, retrieval 
information completely or almost satisfying the retrieval 
conditions using each retrieval information stored in said 

25 retrieval information delivery file and information relating 
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to and a category of the retrieval information, and outputs the 
video information as a result of retrieval according to the 
video index information including the retrieval information. 

45 . The video retrieval system according to Claim 44 ; wherein 
said video information databases, video index information 
databases, retrieval information delivery files, and retrieval 
devices are provided on a network, and 

any retrieval device on the network can be connected via 
a given retrieval information delivery file to any of all of 
the video index information databases and all of the video 
information databases. 

46. The video retrieval system according to Claim 44; wherein 
said video information databases, video index information 
databases, retrieval Information delivery files, and retrieval 
devices are provided on a network, and 

any retrieval device on the network can be connected via 
a given retrieval information delivery file on the network to 
any of all of the video index information databases and all of 
the video information databases. 

47. A video retrieval system comprising: 
a plurality of video information databases each storing 

25 therein a plurality of video information; 
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a plurality of video index information databases each 
storing video index information previously prepared in 
correlation to each video information; 

a retrieval information delivery file generated by 
5 extracting retrieval information for retrieving video 
information stored in said video information databases from the 
video index information stored in said plurality of index 
information databases; and 

a retrieval device for retrieving video information 
10 completely or almost satisfying a desired retrieval condition 
from said video information databases by using said retrieval 
information delivery file; wherein 

the video index information in said video index 
information databases has a tree structure comprising at least 
15 one frame information for managing a given number of successive 
frames in the video information as one unit of information , at 
least one sound information for managing sounds corresponding 
to a given range in the video information as one unit of 
information, at least one segment information for classifying 
20 the frame information and sound information each as one unit 
of information and managing the information as a group having 
desired meanings, and package information for managing a 
plurality of segment information as a group each as a structure 
element object; 

25 the segment information manages package information for 
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managing a plurality of other segm nt information prepared by- 
using the frame information managed by the segment information 
and frames and sounds in the same range as that of the sound 
information as a group, and further the package information is 
5 allocated in addition to the frame information and sound 
information under one segment information in said tree 
structure; 

said retrieval information delivery file stores therein 
the plurality of retrieval information classified into a 

10 plurality of categories, each retrieval Information correlated 
to a structure element object retrieved according to the 
retrieval information and video index information with the 
structure element object present therein; and 

said retrieval device identifies, when retrieval 

15 conditions for a desired video-image are inputted, retrieval 
information completely or almost satisfying the retrieval 
conditions using each retrieval information stored in said 
retrieval information delivery file and information relating 
to and a category of the retrieval information, and outputs the 

20 video information as a result of retrieval according to the 
video index information including the retrieval information. 

48 . The video retrieval system according to Claim 47; wherein 
said video information databases, video index information 
25 databases, retrieval information delivery files, and retrieval 

126 



devices are provided on a network, and 

any retrieval device on the network can be connected via 
a given retrieval information delivery file to any of all of 
the video index information databases and all of the video 
5 information databases. 

49 . The video retrieval system according to Claim 47; wherein 
said video information databases, video index information 
databases, retrieval information delivery files, and retrieval 
10 devices are provided on a network, and 

any retrieval device on the network can be connected via 
a given retrieval information delivery file on the network to 
any of all of the video index information databases and all of 
the video information databases. 
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